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INTRODUCTION 


In  October  1967,  a  Tri-Service  Croup  was  formed  to  investigate 
state-of-the-art  approaches  to  reliability  assessments.  Following  the 
publication  of  the  Tri-Service  Report  (Ref  1),  Picatinny  Arsenal  began 
to  explore  areas  of  use  for  Bayesian  analysis  in  the  reliability  testing  of 
systems.  It  became  obvious  that  for  any  Bayesian  approach  to  be  truly 
useful,  it  must  be  designed  directly  into  the  data  collection  scheme. 
Clearly,  it  would  be  of  little  advantage  to  utilize  an  analysis  which  could 
have  been  effective  for  five  items  if  50  were  actually  tested.  The  savings 
could  only  be  realized  if  the  test  program  itself  was  designed  so  as  to 
only  test  five  items.  In  order  to  achieve  these  savings,  the  SABRE  method 
was  developed.  SABRE  provides  a  method  of  determining  a  reasonable 
sample  size  requirement  for  reliability  testing  of  atomic  projectiles. 


CRITERIA  FOR  DATA  COLLECTION  AND  ANALYSIS 
Assumptions 

It  has  been  assumed  that  the  data  from  each  test  may  be  combined. 

If,  during  actual  testing,  it  is  found  that  the  data  is  not  combinable,  new 
analyses  will  have  to  be  developed  to  reflect  the  reduced  equivalent 
sample  size.  Reliability  analyses  performed  assume  both  component  and 
system  reliabilities  behave  as  random  variables,  distributed  approximately 
as  the  Beta  distribution.  . 

Pa(1-P)p 

for  0£p<J1  and  a,p>-1 

where  R(p/a,p)  is  reliability  as  a  function  of  the  random 
variable  p  given  a  and  p 

a  is  the  equivalent  number  of  successes 

P  is  the  equivalent  number  of  failures 

T  is  the  gamma  function* 

*r  (x+1 )  = j yxe  ydy,  F  (n+1 )  =  n!  neN 
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Analyses  on  these  variables  are  performed  using  the  Tri-Service  Method 
(Ref  1) .  The  general  idea  of  a  Bayesian  approach  has  been  supported 
(Ref  2) .  Finally,  it  is  assumed  that  tne  component  and  system  reliabilities 
are  monotonic  increasing  functions  of  time  at  least  up  until  the  time  of 
production.  Reliability  growth  occurs  both  as  a  function  of  detecting  and 
correcting  deficiencies  and  of  increased  confidence  in  reliability  estimates 
as  more  applicable  data  is  collected. 

Design  of  the  Test  Plan 

The  purpose  of  this  study  was  to  recommend  a  sample  size  for  the 
reliability  test  portion  of  a  test  plan,  with  the  goal  towards  keeping  the 
overall  sample  size  to  a  minimum.  Upon  examination  of  a  test  plan,  it  was 
determined,  for  that  item,  both  the  system  safety  test  rounds  (fuzed)  and 
full  function  test  rounds  would  be  of  proper  configuration  to  provide  suit¬ 
able  data  for  evaluating  reliability.  A  further  examination  utilizing  the 
SABRE  technique,  as  presented,  indicated  that  an  acceptable  reliability 
estimate  could  be  made  well  within  the  bounds  of  the  system  safety  test 
sample  size.  As  a  result,  no  additional  rounds  were  proposed  solely  to 
evaluate  reliability.  A  still  further  investigation  into  the  system  safety 
test  indicated  that,  at  best,  only  a  "gut  feeling"  of  the  system  safety  could 
be  achieved  within  the  limits  of  an  economically  feasible  sample  size.  As 
a  result,  an  alternative  plan  was  proposed  to  incorporate  the  reliability 
test  within  the  system  safety  test,  and  reduce  the  system  safety  test  sample 
size  to  only  that  level  necessary  to  establish  a  reliability  estimate.  The 
feeling  being  that  if  the  reliability  is  sufficiently  demonstrated  to  give 
creditability  to  our  "paper  studies"  of  reliability  then  we  have  justification 
in  assuming  that  the  system  does  indeed  perform  as  anticipated;  thus,  in 
effect,  lending  support  to  our"paper  studies"  of  safety.  While  not  entirely 
scientific,  i.e.  "statistically  valid",  this  procedure  for  evaluating  safety 
is  no  worse  than  the  present  method  of  firing  a  totally  arbitrary  number 
of  rounds  and  postulating  extremely  low  system  premature  rates  from  the 
data.  We  can  indeed  subject  the  test  rounds  to  all  environments  of  interest 
by  using  a  suitably  designed  experiment  and  still  determine  the  effects 
of  each  individual  condition. 


DESIGN  PROCEDURES 


Assumptions 

(a)  Components  and  system  reliabilities  are  Beta  distributed. 
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(b)  Each  component  has  the  same  failure  rate  (attribute) . 

(c)  Excessive  component  failures  will  lead  to  redesign. 

(d)  In  each  test  plan,  fewer  failures  than  those  stated  will  be  seen 
for  each  component. 

(e)  Model  of  system  exists. 

Discussion 

The  subsequent  section  on  Simulation  Procedure  provides  the 
mathematics  used  in  obtaining  a  sample  size  for  estimating  reliability. 

That  procedure  consists  of  a  rather  simple  and  striaghtforward  analytical 
approach.  Simply  put,  it  was  assumed  a  certain  experiment  was  performed 
(i.e. ,  a  given  test  plan  adopted)  and  that  certain  data  was  collected. 

Then  an  analysis  using  that  data  was  performed  (i.e.,  a  system  posterior 
was  calculated) .  The  results  of  this  analysis  were  examined  and,  if  suit¬ 
able,  then  different  data  was  assumed  for  the  same  experiment  sample  size. 
If  the  posterior  was  unsuitable,  then  a  different  experiment  was  postulated. 
The  end  result  is  an  "optimal  experiment",  i.e.,  one  producing  the  best 
posterior  over  a  range  of  possible  experimental  outcomes.  A  graph  is 
included  to  indicate  some  of  the  experiments  and  outcomes  which  were 
examined.  A  brief  explanation  of  the  various  assumptions  may  now  be  of 
help.  The  assumptions,  needless  to  say,  form  the  basis  for  the  procedure 
and,  as  such,  deserve  individual  examination.  Item  (a)  assumes  a  Beta 
fit  can  be  found  for  each  set  of  data.  This  is  a  rather  general  assumption 
about  the  property  of  components,  and  with  the  exception  of  all  but  the 
most  wildly  misbehaving  data  a  good  approximation  to  reality  for  those 
systems  considered.  Also  assumed  by  item  (a)  is  that  the  system  reli¬ 
ability  is  Beta-distributed.  This  again  is  a  reasonable  assumption,  bar¬ 
ring  any  evidence  to  the  contrary  (i.e.,  bi-  or  tri-modal  distribution 
caused  by  multiple  manufacturers,  etc) .  Assumptions  (b),  and  (c) ,  and 
(d)  form  a  limiting  case  of  the  considerations  and,  as  such,  pose  a  condi¬ 
tion  for  rejecting  a  design/system  as  unsuitable.  Assumption  (c)  is  an 
outcome  of  the  necessity  for  using  component  level  information  to  make 
system  level  statements.  Without  a  model  of  the  system,  based  upon  the 
components,  this  procedure  fails.  As  a  result,  the  system  estimates  can 
only  be  as  accurate  as  the  model  being  used.  This  area  will,  in  fact,  be 
one  for  close  future  surveillance  to  assure  that  the  model  is  always 
representative  of  the  system  as  fielded.  The  simulation  procedure  shown 
is  a  very  simple  Monte  Carlo  approach.  The  only  areas  worth  noting  are: 
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(a)  The  conservative  prior  of  a  =  0,  p  =  0  is  used  on  the  component 
level  pending  the  outcome  of  the  engineering  design  (ED)  phase  testing 
when  a  less  conservative  prior  can  be  determined.  The  effect  is  to  possibly 
require  a  slightly  higher  sample  size  at  that  time  than  may  ultimately  be 
required. 

(b)  The  number  of  simulations  stated,  200,  is  purely  arbitrary  at 
this  time,  decided  on  a  computer  cost  basis.  Before  each  test  phase,  a 
finalized  test  program  will  be  determined  as  indicated  in  above  paragraph. 
Then,  a  sufficiently  large  number  (N)  of  Monte  Carlo  simulations  will  be 
used  to  make  the  results  insensitive  to  such  N. 

Simulation  Procedure 


Using  engineering  judgment,  pick  the  highest  failure  rate/component. 
Suppose,  in  a  sample  of  40  components,  we  will  have  one  component  failure. 
Our  Beta  parameters  for  the  component  reliability  distribution  are: 

a  =  #  successes  =  39 


P  =  #  failures  =  1 

These  parameters  are  sufficient  to  describe  a  Beta  curve 
p  fD>  _  r* (g+p+2) _  RaM-Ri^ 

Fp(R)  "  r(a+nr(p+n  R  (1  R) 

The  Beta  curve  looks  like: 


Mean  =  (a+1)/  (a+p+2) 

Mode  =  a/  (a+p) 

For  each  block  in  the  system  Model  generate  a  random  variate  from 
Fp(R/a,p). 
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Having  a  numerical  value  in  each  block  of  the  system  model,  calculate 
a  point  estimate  for  system  success.  This  can  be  done  with  a  system  suc¬ 
cess  equation,  a  system  reliability  computer  program,  or  a  failure  equation. 
Storing  this  point  estimate,  repeat  the  process.  Generate  random 
variates  for  components  and  calculate  a  system  estimate.  Continue  this 
procedure  until  200  system  point  estimates  have  been  compiled.  Calculate 
the  mean  and  standard  deviation  of  these  200  values  and  calculate  the 
equivalent  system  Beta  parameters  a  and  p .  The  mode  is  the  best  estimate 
of  system  reliability.  In  certain  cases,  where  the  equivalent  number  of 
system  failures  is  less  than  zero,  the  Beta  curve  looks  like  the  following: 


0  R  1 


The  curve  is  asymptotic  to  R  =  1 .00.  In  this  case,  the  mode  is  invalid  and 
the  mean  must  be  used  as  a  best  estimate  for  the  system  reliability.  After 
the  equivalent  system  parameters  and  best  estimate  are  computed,  the  90% 
confidence  value  can  be  calculated.  This  is  done  by  finding  the  point  on 
the  axis  which  90%  of  the  area  under  the  curve  falls  to  the  right  of.  We 
now  have  the  following  information: 

(a)  Component  sample  size  and  failure  rate. 

(b)  Best  estimate  of  system  reliability. 

(c)  90% confidence  value  for  system  reliability. 

It  is  now  necessary  to  review  these  values  and  see  if  they  meet  the  require¬ 
ments.  If  these  values  are  too  low,  then  it  may  be  necessary  to  increase 
our  sample  size  tested  and/or  reduce  the  number  of  component  failures 
which  are  acceptable.  Either  of  these  actions  will  shift  the  curve  to  the 
right,  increasing  the  best  estimate  and  90%  confidence  value.  Figure  1 
gives  an  example  of  various  possible  results  for  a  range  of  test  plans. 


EXAMPLE  OF  APPROACH 

This  section  provides  an  example  of  how  data  may  be  analyzed.  For 
simplicity,  the  conservative  a=0,  p=0  prior  was  again  used;  however,  a 
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less  conservative  prior  could  be  applied  if  applicable  prior  test  data  exists. 
In  this  example,  a  total  of  994  components  are  tested  as  parts  of  systems, 
with  six  component  failures  occurring.  In  all  likelihood,  corrective  action 
would  be  taken  to  preclude  the  recurrence  of  failures  similar  to  those  six 
which  occurred.  However,  barring  a  total  major  system  redesign,  all 
six  failures  should  be  included  as  indicative  of  other  failures  which  po¬ 
tentially  exist  and  are  at  present  uncorrected.  At  no  time  are  these  statis¬ 
tical  procedures  meant  to  outweigh  or  replace  sound  engineering  judgment, 
and  each  failure  which  occurs  should  be  examined,  not  only  for  its  impact 
on  the  system  but,  more  importantly,  how  strongly  this  failure  contradicts 
what  were  previously  held  "truths"  about  the  manner  in  which  the  system 
operates.  Bearing  this  in  mind,  the  statistical  procedure  is  performed 
once  the  engineers  are  convinced  that  they  can  continue  to  accent  their 
"prior  beliefs". 

Let  us  say  we  have  actually  tested  45  safety  test  rounds,  18  full 
function  rounds,  and  90  firing  table  rounds.  This  gives  us  a  total  sample 
size  of  153  rounds.  This  should  give  306  tests  of  double-redundant  com¬ 
ponents  and  459  tests  of  triple-redundant  items.  Say,  for  example,  we 
use  a  system  as  below: 


And  say  our  153  rounds  result  in  three  failures  for  p,  one  failure  for  BT, 
and  two  failures  for  TS,  with  11  system  no  tests  (no  TM  at  all) .  We  have 
test  results  below: 


COMPONENT 

SAMPLE  SIZE 

SUCCESS 

FAILURE  (b) 

BT 

284 

283 

1 

P 

426 

423 

3 

TS 

284 

282 

2 
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We  construct  cumulative  density  functions  for  the  Beta  distributions 
of  each  component  using  the  a  and  p  parameters  as  above  (roughly  pictured 
below) . 


T  1.0 

I 

|  rc f  =  p(/X  R) 

i 

-  0 

Component  10 


We  will  then  randomly  sample  along  the  CDF  axis  to  get  a  correspond¬ 
ing  value  of  R  for  each  of  the  blocks  in  the  system  diagram  (1)  and  calcu¬ 
late  one  point  estimate  for  system  reliability.  Repeating  the  sampling  pro¬ 
cedure  many  times,  we  will  calculate  a  large  number  of  system  point 
estimates  (like  200)  and  plot  a  histogram  of  these  points.  We  can  then  take 
a  Beta  fit  to  this  histogram  and  calculate  any  desired  statistics  for  this 
posterior  system  Beta  distribution  as  required. 
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SUMMARY 


The  result  of  one  study  was  the  recommendation  for  a  substantial 
reduction  in  the  size  of  the  system  safety  portion  of  the  test  program. 

This  reduction  was  brought  about  through  the  judicious  use  of  component 
level  reliability  test  data  for  determining  system  reliability.  While  a 
numerical  safety  estimate  was  not  made  based  upon  test  data,  an  engineer¬ 
ing  analysis  of  system  safety  would  be  made  from  all  available  data.  This 
engineering  analysis  would  depend  in  large  part  on  the  belief  that  the 
system  operates  as  expected;  and  that  belief  is  substantiated  to  a  degree 
by  the  reliability  data  taken. 

The  SABRE  method  is  simply  an  analytical  approach  to  reliability 
test  program  design.  Various  experiments  are  assumed,  worst  case  data 
is  *hen  assumed,  and  experimental  results  calculated  by  the  techniques 
described.  The  outcomes  are  screened  to  determine  the  optimum  experi¬ 
ment  yielding  desired  outcomes.  Key  points  in  the  analytical  procedure 
are  formulation  of  priors,  Monte  Carlo  analysis  to  yield  system  posteriors, 
and  a  risk  analysis  using  the  parameters  of  the  posterior  distribution. 
Imbedded  in  the  procedures  is  a  reliance  on  a  mathematical  model  for  the 
system.  A  computer  routine  "SABRE"  soon  to  be  available  from  Picatinny 
Arsenal  will  utilize  the  most  advanced  modeling  techniques  yet  available; 
'Alt  that  we  will  save  for  future  discussion. 
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